just to get a feeling
Changing the power of the input signal does not change anything.
In your received signal, there seem to be high-power peaks. How large is the digital signal amplitude during these?
Since the mirrored peak seems to be just a bit weaker, my suspicion still is too high input power.
If, and only if my suspicion is correct the problem is the inherent nature of semiconductor amplifiers, and the awesomeness that is math:In any case I can't understand why the replica is there (?).
Your amplifier has the job of taking the input signal amplitude x and map it to the output y, effectively making it a function
mathematically
,
being the
amplification. In reality,
is better
represented as an exponential function. However, using the power of
Taylor, and a bit of analysis, without loss of generality, we can
represent that as a power series:
Amplifier producers try to keep
as small as technically possible.Obviously, we want all
to be as
close to 0 as possible, but for 
, where
it represents the linear amplification we want; 
would then collapse into the function we want. Hence, we want to
truncate the series after the 2nd element.
usually
isn't as bad for RF signals, it's "just" DC offset, something
relatively easy to cancel (e.g. by capacitive coupling or other
means of high-pass filtering).Remember that I mentioned that we need to apply Taylor's Theorem to the exponential, which develops the original function into a series around a specific point, ie. if we truncate the series for
,
because we trust the manufacturer and want to use the amplifier just
as it was actually (instead of approximately)
linear, that's only a valid thing to do if we don't drive the
amplifier with 
that are
larger than what the manufacturer promised to amplify linearly;
obviously, for sufficiently large , 
and
higher powers can get substantially bigger than 
, and
then you see interesting behaviour. (you'll notice that when looking
at amplifier datasheets, their functions typically look a bit like a
quadratic with negative coefficient for
-- that's actually the case for semiconductors, usually).Now, that will distort your ampitude, but you see frequency magic. That's only natural, however: If you look at this,
contains a
term.
Assume furthermore that 
.
Trigonometrics has it that
so that's where a component with twice the frequency comes into your
, additional
to the "regular" amplified original signal, so we can say:
,subsuming all the third order and above terms as well as DC in
.Now, the signal with the "fake" 2f component gets fed into a mixer. A mixer is just an amplifier where the manufacturer did his best to keep all
small but
, so to
enable people to use the squaring to produce its output 
; in our
case, we get funky frequencies. Remember that 
is our
input frequency, and let 
be
our local oscillator, ie. the center frequency that becomes 0Hz in
baseband:
That formula is going to be fun to expand, calculate the powers of the trigonometric functions etc.
However, the gist is that you'll simply have to assume that now all frequencies mix with their multiple, so the result doesn't only contain
, and 
(that being what we want to have in baseband!), but also 
and 
,
which is possibly the mirror you're seing in complex baseband.Best regards,
Marcus