Could my business provide financial services without becoming authorised or registered as a payment service provider?

Yes, using a Banking-as-a-Service provider means pretty much any business has the potential to distribute financial services using their own-branded products.