I'm thinking yes for one basic reason. Reading a book takes hours, and for something like that, you want a good, comfortable experience— no eye strain (well, no more than with a paper book), a reasonable page size, ease of use— as well as the advanced functions like annotations, word look up, and searching.
Certainly, it's possible, even likely, that more folks have the Kindle for iPhone app than have a Kindle. Whether they routinely read long term on their iPhones is another question. The Kindle for iPhone app is easy to use and intuitive, but the main impetus behind its popularity is the popularity of the iPhone itself. The user who already owns the iPhone (or iTouch) can download the app for no extra cost, and it's a device he will always have with him, so why not use it? For casual readers who read thirty minutes a day on the bus or subway, that might be fine. But for folks who want to put their heads down and not come up for air until they finish the book, that might not be so fine.
One thing that came to light lately was that the Kindle DX was not working well for some Princeton students because it didn't fit their needs in terms of highlighting, browsing, and citing page numbers (although some of the discontent might have come from a lack of training; students didn't seem to know how to do all the things the kindle can do).
Which, I think, actually makes my point. The current crop of eReaders work great for reading novels and nonfiction books that aren't illustrated. Making a device that does everything everyone could want is going to be a lot harder. And with competition (Kindle, Nook, iRex, Sony, Alex, eDGE, Cool-er, etc.) prices will come down, making a dedicated eReader more affordable. I think more folks are going to want to curl up with something larger than an iPhone when they sit down for a good, long read.