What kind of graphics hardware do you have? the error message seems to indicate that your video card/chip doesn't have adequate vertex and/or pixel shader support.
A painfully old card which supports DirectX7-related-stuff only. That means no vertex/pixel shader support of any kind. You can stop trying to get Demul (or any other Dreamcast emulator for that matter) to work - it won't happen.
I'm not sure but nullDC could handle it if the CPU is something like an Athlon 3200 or Intel Core or even something newer. If it's something like a Celeron or a Pentium 4 the load would be too much.
Chankast could handle it too in that case.