Investing in the small-cap oil sector is a risky business. To be successful, you have to be adept at balancing risk and reward.

Most small-cap oil companies don’t succeed, but those that do can generate huge returns for early investors if everything goes to plan. Finding these opportunities is the key. Investments where you can make 10 or 20 times your initial investment in the best case are worth taking a bet on. Of course, not every opportunity like this will work out but those that do more than make up for those that fail.

Pantheon Resources (LSE: PANR) and Rockhopper Exploration (LSE: RKH) are two such companies. Both have the assets in place to generate impressive returns for shareholders over the long-term, but at current prices is it worth getting involved? Does the potential reward make up for the risk taken on?

Double again? 

As the rest of the oil sector has languished, shares in Pantheon have charged higher this year. 

Year-to-date shares in the company are up by 260% as the firm has reported multiple successes on its acreage in Polk County, Texas. However, alongside the successes, there have also been failures. At the beginning of September shares in the company dropped by more than 40% after the company announced that its second well in Texas had to be plugged and abandoned. This dramatic markdown of Pantheon’s shares made it clear that the value of the company may have been too generous for the risk still associated with it. 

Assuming everything goes to plan going forward, City analysts expect the company to report earnings per share of 4.6p for the year ending 30 June 2017, which implies a valuation of 22.2 times forward earnings. This rich valuation coupled with the exploratory work Pantheon still has ahead of it signals that there’s plenty of risk ahead for the company’s shares and as a result, the potential upside may be limited.

Terrible year but plenty of potential 

Compared to Pantheon, shares in Rockhopper have had a downright terrible year. During the past 12 months shares in the company have lost a third of their value, but at current levels they may be attractive for long-term investors.

Rockhopper’s value is tied up in the company’s assets, specifically in the Sea Lion Complex. It’s thought that this asset contains 517 mmbbl of resources and has a break-even cost of $45 per barrel. So, even in the current environment where oil prices are struggling to move above $50 a barrel, Sea Lion will be profitable. 

City analysts estimate that Rockhopper’s net asset value stands at around 93p per share, more than 200% above current levels. There’s $63m of cash on the balance sheet as well to protect against the downside. What’s more, if the oil price returns to $70 or $80 a barrel this net asset value could see significant upgrades. In other words, Rockhopper has the sort of attractive risk/reward profile small-cap oil investors require. 

Overall, Rockhopper looks to be the better investment of these two early stage small-cap producers. 

The worst mistake you could make

According to a study conducted by financial research firm DALBAR, the average investor realised an average annual return of only 3.7% a year over the past three decades, underperforming the wider market by around 5.3% annually.

This underperformance can be traced back to several key mistakes that all investors make. To help you realise and understand the most common mis-steps, the Motley Fool has put together this new free report entitled The Worst Mistakes Investors Make.

The report is a collection of Foolish wisdom, which should help you avoid needlessly losing too many more profits. Click here to download your copy today.

Rupert Hargreaves has no position in any shares mentioned. The Motley Fool UK has no position in any of the shares mentioned. We Fools don't all hold the same opinions, but we all believe that considering a diverse range of insights makes us better investors.