Should a custom Splunk app use a separate index by default?

I would generally recommend that data go to the system default (i.e., main) index by default. There is a significant environment-specific and system-level configuration for defining an index (disk space, volumes, retention policy, etc), and by specifying a new one, you would be significantly complicating the (correct) installation of your application. Separate identification of your data can be accomplished with distinct sourcetypes.

A separate index is important if you need to be able to move, delete, age, archive (etc) this data differently from the rest of the data in Splunk. You can and should think about doing this, but I would say that you should think about it separately from installation and configuration of the app. If necessary, it should be done before app installation. You can define these parameters for the index, and then tell your app inputs to use that index. But you should not force this by default.