A portfolio of high net worth clients may represent the holy grail for financial advisers, but new data suggests they are a rarer breed than most advisers imagine with many taking their advice from private bankers and other sources.