Dan,
It is possible but I doubt it is a good idea. There are a number of problems with rectangular ducts. First, you would need to make smooth transitions between the round and rectangular ducts. That is of course possible but not simple. Secondly, assuming like materials, a rectangular duct exhibits higher losses than a round duct. The equivalent hydraulic diameter of a rectangular duct is twice the product of width and height divided by the sum of width and height. So a square duct needs to be 6" on a side to equal a 6" diameter duct (all inside numbers). If the width is twice the height a 4.5" by 9"rectangular duct is needed to match a 6" diameter. More and more cross-sectional area is needed as the duct becomes more oblong. The air speed is minimum near the corners so dust may collect at the corners of a rectangular duct and this effect gets worse as the duct becomes more oblong. Finally, a rectangular duct needs to be made stronger (heavier metal) than a round duct to withstand the negative pressure. And of course that gets worse as the duct gets more oblong. We would welcome some more details of your layout.
Bababrown