Angle brackets in non-math-mode text

You could employ the commands textlangle and textrangle provided by the textcomp package. If that’s too many letters to type each time, you could define shortcuts for them, say la and ra. The following MWE illustrates this idea, also

% !TEX TS-program = xelatex
documentclass{article}
usepackage{fontspec}
% setmainfont{Linux Libertine O} % select your main text font here
usepackage{textcomp,    % for textlangle and textrangle macros
            xspace}
newcommandla{textlanglexspace}  % set up short-form macros
newcommandra{textranglexspace}
begin{document}
noindent
textlangle and textrangle; newline 
la and ra.   % abbreviated macros; note: no need to insert explicit whitespace after "la"
end{document}

enter image description here

fontspec doesn’t have an interface for substituting some characters with characters from a different font, but thankfully egreg provided a package to do just that:

documentclass{article}
usepackage{fontspec}
usepackage{newunicodechar}
setmainfont{Linux Libertine O}

newfontfamilyanglefont{Free Serif}    % Change to whatever font you prefer that has ⟨⟩.
newunicodechar{⟨}{{anglefont ⟨}}
newunicodechar{⟩}{{anglefont ⟩}}

begin{document}
This is Linux Libertine with ⟨ and ⟩ from {anglefont Free Serif}.
end{document}

result