When the payment provider (WorldPay in my case) calls the callback URL to submit a payment notification to my site, it has no session so commerce_checkout_access returns FALSE causing the payment provider to see a 404 error. What is the recommended approach to get around this?
The main thing to determine is exactly what your payment provider needs to do when it pings your site. Typically they're just sending payment notifications that you should then validate and process, resulting in payment transactions being created for your orders. This shouldn't have to depend on the checkout URLs at all.
In the PayPal module, I use a separate "IPN listener" menu item that is always accessible (and therefore session independent). PayPal gives you several ways to validate an IPN is legitimate before processing it, so make sure you don't skip this step for your gateway.
To process the transaction, you're going to need the payment method instance that was selected during checkout to submit payment. I get this in the PayPal module by including the instance ID in the IPN URL that PayPal POSTs its data to. If you can't do that for some reason, then you can load the order and find the selected payment method instance in the $order->data array.
For Drupal 6 and Ubercart I had to modify things so that it would return a simple "payment complete" page that gets served via WorldPay's servers (which also strip out HTML comments, scripts, etc.) that includes a note saying "Please wait while we re-direct you back to the store" and a META redirect to the Ubercart completion page back on the store domain with a two-second delay.
You could perhaps use WorldPay's resultY.html page if you are able to add a META redirect to that.
There is apparently no way to get WorldPay to redirect properly back to the store site (i.e. with an HTTP Location: header) so you have to fake it with a META redirect. This is WorldPay's suggested solution.
Comments
see below!