I did it in one of my projects.
Following are the steps I followed: 
1. Add a field called wholesale_price to the product.
2. Add a role called wholeseller (only if you want to set price by role)
3. Create a product pricing rule:
When calculating the sell price of an order
Condition: Entity has field => Entity: commerce-line-item:commerce_product
Condition: Entity has field => Entity: commerce-line-item:commerce-product: field_wholesale_price
opt A. (to check by roles)
Condition: User has roles: site:current-user, Roles: wholeseller
-- or --
opt B. (to check by user id)
Condition: Add new condition: site:current-user:uid, whateveruserid
Action: Commerce Line Item: Set the unit price to a specific amount: Parameter: Line item: [commerce_line_item], Value: [commerce-line-item:commerce-product:field-adminprice:amount]
Following is the Export of Rule:
{ "rules_wholesale_price_for_wholesellers_of_accessories" : {
    "LABEL" : "Wholesale Price For Wholesellers if has wholesaleprice field",
    "PLUGIN" : "reaction rule",
    "REQUIRES" : [ "rules", "commerce_line_item", "commerce_product_reference" ],
    "ON" : [ "commerce_product_calculate_sell_price" ],
    "IF" : [
      { "entity_has_field" : { "entity" : [ "commerce-line-item" ], "field" : "commerce_product" } },
      { "entity_has_field" : {
          "entity" : [ "commerce-line-item:commerce-product" ],
          "field" : "field_wholesale_price"
        }
      },
      { "user_has_role" : {
          "account" : [ "site:current-user" ],
          "roles" : { "value" : { "6" : "6" } },
          "operation" : "OR"
        }
      }
    ],
    "DO" : [
      { "commerce_line_item_unit_price_amount" : {
          "commerce_line_item" : [ "commerce_line_item" ],
          "amount" : [ "commerce-line-item:commerce-product:field-wholesale-price:amount" ],
          "component_name" : "base_price",
          "round_mode" : "1"
        }
      }
    ]
  }
}
note : you have to change the value in  "roles", which is 6 in my case
I hope this helps.
Comments
Unfortunately my situation is more complicated as it's more a case of each user having having their own personal price, so for example there could be 100 customers each with their own unique price for a certain product.
This is why I was thinking that a separate entity that links to the user and the product with the custom price would be the best way. I just can't work out a rule that would fetch the correct entity an then apply the price.
Hi,
My situation was almost the same as yours, we created a rule useing option (b)
opt B. (to check by user id)
Condition: Add new condition: site:current-user:uid, whateveruserid
And we keep cloning the rule and changing the user id for every new customer.
It may sound a little complicated but it is very easy manage and takes less than a minute for every new customer.