Having tried out Inkscape and Scribus, I must say that Inkscape's UI is much more convenient to work with.

So my question is, would it be a good idea to instead improve Inkscape for use in desktop publishing?
Both artistic and publications uses require much of the same features. It seems to me that using a single program would produce a better experience for users.
Off the top of my head, my missing features comparison between Inkscape and Scribus is:

Inkscape
1. Color Support (yes, this is a big one)
2. Multipage support
3. PDF X/3 Export
4. Easier working with clip paths, get rid of clip path bug

Scribus
1. UI (I find Inkscape's design better)
2. Image support (load-everything-to-memory and lack of caching can cause problems)

Of the inkscape missing features, I believe 1&2 are already planned at some point in the future. What do you think of implementing these features and then making an extension/patchset/fork that provided raster image management? I understand that Inkscape is designed to be an SVG editor, but I feel that it could be used for much more.

Most of the features you want for Inkscape are acknowledged. Print support will be improved as the SVG standard improves.

SVG was originally designed for display in a Web browser, not print. So print features are still coming.