coyotekyk
Your curiosity on how to get a smooth flying rotor is commendable. I am by no means an expert but I have developed a pretty good understanding over the years and I'll try to pass on some of what I have learned. Forgive me if you already know some or all of what I say.
At the entry-level of understanding, an unbalanced, shaking rotor must have some mechanical change made to get it smooth. Among the list of things a person can do are...1) Dial-indicate and/or weight-match everything that rotates 2) Add weights to the rotating assembly somewhere 3) Twist or Change the Angle of Incidence of one or more blades 4) Add bendable trim tabs to the trailing edges of blades at some distance from the tip 5) Move the chordwise CG by adding leading edge or trailing edge weight or by carefully painting using paint weight to achieve the same thing.
There's more on the list but these are the most common things that folks usually measure and change.
All balancing attempts must have a means to take repeatable, consistent vibrations which I think you already have.
If you measure your vibrations in flight at the speed where you spend most of your time, and make an adjustment to ONE of the items in the above list and then go fly and measure again, you will gain experience and understanding about what adjustment fixes what vibration. If you are very lucky, someone has already done this for a machine just like yours and you don't have to start from scratch.
Most balancers or software packages use one or more accelerometers to measure magnitude of your vibe and either photcell pickup or Hall effect trigger to measure clock position where the vibration happens. The clock position can be like 1:00 -12:00 or 0-360 or 1:00 - 24:00, etc.
As an example to help you understand, Let's say...looking down on your spinning rotor from above, we draw a large circle that represents the rotor disk and we pencil-in the top view of a simple fuselage, fixed to the center of the circle. If we assume that 12:00 is the nose and 6:00 is the tail, that can be the seeds to label the sketch accurately around the 360 degrees. Now lets say that the big circle gets a number 1 and the center of rotation (mast) gets a zero. Divide this with 9 concentric circles. The smallest is 0.1, the next 0.2, etc. as you progress outward. These circles represent magnitude of your vibe and 1:00 -12:00 represent azimuth or clock position.
Now you can plot data points from your electronic balancer and change ONE thing at a time from paragraph 2. This moves your datapoint and you can begin to get a "feel" for what needs to change to get smoother. More later.