Re: [PATCH v2 1/6] ieee1275: request memory with ibm, client-architecture-support

[Daniel Kiper]

On Thu, Dec 01, 2022 at 04:11:56PM -0500, Stefan Berger wrote:
> From: Daniel Axtens <dja@axtens.net>
>
> On PowerVM, the first time we boot a Linux partition, we may only get
> 256MB of real memory area, even if the partition has more memory.
>
> This isn't enough to reliably verify a kernel. Fortunately, the Power
> Architecture Platform Reference (PAPR) defines a method we can call to ask
> for more memory: the broad and powerful ibm,client-architecture-support
> (CAS) method.
>
> CAS can do an enormous amount of things on a PAPR platform: as well as
> asking for memory, you can set the supported processor level, the interrupt
> controller, hash vs radix mmu, and so on.
>
> If:
>
>  - we are running under what we think is PowerVM (compatible property of /
>    begins with "IBM"), and
>
>  - the full amount of RMA is less than 512MB (as determined by the reg
>    property of /memory)
>
> then call CAS as follows: (refer to the Linux on Power Architecture
> Reference, LoPAR, which is public, at B.5.2.3):
>
>  - Use the "any" PVR value and supply 2 option vectors.
>
>  - Set option vector 1 (PowerPC Server Processor Architecture Level)
>    to "ignore".
>
>  - Set option vector 2 with default or Linux-like options, including a
>    min-rma-size of 512MB.
>
>  - Set option vector 3 to request Floating Point, VMX and Decimal Floating
>    point, but don't abort the boot if we can't get them.
>
>  - Set option vector 4 to request a minimum VP percentage to 1%, which is
>    what Linux requests, and is below the default of 10%. Without this,
>    some systems with very large or very small configurations fail to boot.
>
> This will cause a CAS reboot and the partition will restart with 512MB
> of RMA. Importantly, grub will notice the 512MB and not call CAS again.
>
> Notes about the choices of parameters:
>
>  - A partition can be configured with only 256MB of memory, which would
>    mean this request couldn't be satisfied, but PFW refuses to load with
>    only 256MB of memory, so it's a bit moot. SLOF will run fine with 256MB,
>    but we will never call CAS under qemu/SLOF because /compatible won't
>    begin with "IBM".)
>
>  - unspecified CAS vectors take on default values. Some of these values
>    might restrict the ability of certain hardware configurations to boot.
>    This is why we need to specify the VP percentage in vector 4, which is
>    in turn why we need to specify vector 3.
>
> Finally, we should have enough memory to verify a kernel, and we will
> reach Linux. One of the first things Linux does while still running under
> OpenFirmware is to call CAS with a much fuller set of options (including
> asking for 512MB of memory). Linux includes a much more restrictive set of
> PVR values and processor support levels, and this CAS invocation will likely
> induce another reboot. On this reboot grub will again notice the higher RMA,
> and not call CAS. We will get to Linux again, Linux will call CAS again, but
> because the values are now set for Linux this will not induce another CAS
> reboot and we will finally boot all the way to userspace.
>
> On all subsequent boots, everything will be configured with 512MB of RMA,
> so there will be no further CAS reboots from grub. (phyp is super sticky
> with the RMA size - it persists even on cold boots. So if you've ever booted
> Linux in a partition, you'll probably never have grub call CAS. It'll only
> ever fire the first time a partition loads grub, or if you deliberately lower
> the amount of memory your partition has below 512MB.)
>
> Signed-off-by: Daniel Axtens <dja@axtens.net>
> Signed-off-by: Stefan Berger <stefanb@linux.ibm.com>
> ---
>  grub-core/kern/ieee1275/cmain.c  |   3 +
>  grub-core/kern/ieee1275/init.c   | 165 +++++++++++++++++++++++++++++++
>  include/grub/ieee1275/ieee1275.h |   8 ++
>  3 files changed, 176 insertions(+)
>
> diff --git a/grub-core/kern/ieee1275/cmain.c b/grub-core/kern/ieee1275/cmain.c
> index 4442b6a83..b707798ec 100644
> --- a/grub-core/kern/ieee1275/cmain.c
> +++ b/grub-core/kern/ieee1275/cmain.c
> @@ -123,6 +123,9 @@ grub_ieee1275_find_options (void)
>             break;
>           }
>       }
> +

#if defined(__powerpc__) ?

> +      if (grub_strncmp (tmp, "IBM,", 4) == 0)
> +     grub_ieee1275_set_flag (GRUB_IEEE1275_FLAG_CAN_TRY_CAS_FOR_MORE_MEMORY);

#endif

>      }
>
>    if (is_smartfirmware)
> diff --git a/grub-core/kern/ieee1275/init.c b/grub-core/kern/ieee1275/init.c
> index 2adf4fdfc..0bc571e3e 100644
> --- a/grub-core/kern/ieee1275/init.c
> +++ b/grub-core/kern/ieee1275/init.c
> @@ -200,11 +200,176 @@ heap_init (grub_uint64_t addr, grub_uint64_t len, 
> grub_memory_type_t type,
>    return 0;
>  }
>
> +/*
> + * How much memory does OF believe it has? (regardless of whether
> + * it's accessible or not)
> + */
> +static grub_err_t
> +grub_ieee1275_total_mem (grub_uint64_t *total)
> +{
> +  grub_ieee1275_phandle_t root;
> +  grub_ieee1275_phandle_t memory;
> +  grub_uint32_t reg[4];
> +  grub_ssize_t reg_size;
> +  grub_uint32_t address_cells = 1;
> +  grub_uint32_t size_cells = 1;
> +  grub_uint64_t size;
> +
> +  /* If we fail to get to the end, report 0. */
> +  *total = 0;
> +
> +  /* Determine the format of each entry in `reg'.  */
> +  if (grub_ieee1275_finddevice ("/", &root))
> +    return grub_error (GRUB_ERR_UNKNOWN_DEVICE, "couldn't find / node");
> +  if (grub_ieee1275_get_integer_property (root, "#address-cells", 
> &address_cells,
> +                                       sizeof (address_cells), 0))
> +    return grub_error (GRUB_ERR_UNKNOWN_DEVICE, "couldn't examine 
> #address-cells");
> +  if (grub_ieee1275_get_integer_property (root, "#size-cells", &size_cells,
> +                                       sizeof (size_cells), 0))
> +    return grub_error (GRUB_ERR_UNKNOWN_DEVICE, "couldn't examine 
> #size-cells");
> +
> +  if (size_cells > address_cells)
> +    address_cells = size_cells;
> +
> +  /* Load `/memory/reg'.  */
> +  if (grub_ieee1275_finddevice ("/memory", &memory))
> +    return grub_error (GRUB_ERR_UNKNOWN_DEVICE, "couldn't find /memory 
> node");
> +  if (grub_ieee1275_get_integer_property (memory, "reg", reg,
> +                                       sizeof (reg), &reg_size))
> +    return grub_error (GRUB_ERR_UNKNOWN_DEVICE, "couldn't examine 
> /memory/reg property");
> +  if (reg_size < 0 || (grub_size_t) reg_size > sizeof (reg))
> +    return grub_error (GRUB_ERR_UNKNOWN_DEVICE, "/memory response buffer 
> exceeded");
> +
> +  if (grub_ieee1275_test_flag (GRUB_IEEE1275_FLAG_BROKEN_ADDRESS_CELLS))
> +    {
> +      address_cells = 1;
> +      size_cells = 1;
> +    }
> +
> +  /* Decode only the size */
> +  size = reg[address_cells];
> +  if (size_cells == 2)
> +    size = (size << 32) | reg[address_cells + 1];
> +
> +  *total = size;
> +
> +  return grub_errno;
> +}
> +
> +#if defined(__powerpc__)
> +
> +/* See PAPR or arch/powerpc/kernel/prom_init.c */
> +struct option_vector2
> +{
> +  grub_uint8_t byte1;
> +  grub_uint16_t reserved;
> +  grub_uint32_t real_base;
> +  grub_uint32_t real_size;
> +  grub_uint32_t virt_base;
> +  grub_uint32_t virt_size;
> +  grub_uint32_t load_base;
> +  grub_uint32_t min_rma;
> +  grub_uint32_t min_load;
> +  grub_uint8_t min_rma_percent;
> +  grub_uint8_t max_pft_size;
> +} GRUB_PACKED;
> +
> +struct pvr_entry
> +{
> +  grub_uint32_t mask;
> +  grub_uint32_t entry;
> +};
> +
> +struct cas_vector
> +{
> +  struct
> +  {
> +    struct pvr_entry terminal;
> +  } pvr_list;
> +  grub_uint8_t num_vecs;
> +  grub_uint8_t vec1_size;
> +  grub_uint8_t vec1;
> +  grub_uint8_t vec2_size;
> +  struct option_vector2 vec2;
> +  grub_uint8_t vec3_size;
> +  grub_uint16_t vec3;
> +  grub_uint8_t vec4_size;
> +  grub_uint16_t vec4;
> +} GRUB_PACKED;
> +
> +/*
> + * Call ibm,client-architecture-support to try to get more RMA.
> + * We ask for 512MB which should be enough to verify a distro kernel.
> + * We ignore most errors: if we don't succeed we'll proceed with whatever
> + * memory we have.
> + */
> +static void
> +grub_ieee1275_ibm_cas (void)
> +{
> +  int rc;
> +  grub_ieee1275_ihandle_t root;
> +  struct cas_args
> +  {
> +    struct grub_ieee1275_common_hdr common;
> +    grub_ieee1275_cell_t method;
> +    grub_ieee1275_ihandle_t ihandle;
> +    grub_ieee1275_cell_t cas_addr;
> +    grub_ieee1275_cell_t result;
> +  } args;
> +  struct cas_vector vector =
> +  {
> +    .pvr_list = { { 0x00000000, 0xffffffff } }, /* any processor */
> +    .num_vecs = 4 - 1,
> +    .vec1_size = 0,
> +    .vec1 = 0x80, /* ignore */
> +    .vec2_size = 1 + sizeof (struct option_vector2) - 2,
> +    .vec2 = {
> +      0, 0, -1, -1, -1, -1, -1, 512, -1, 0, 48
> +    },
> +    .vec3_size = 2 - 1,
> +    .vec3 = 0x00e0, /* ask for FP + VMX + DFP but don't halt if unsatisfied 
> */
> +    .vec4_size = 2 - 1,
> +    .vec4 = 0x0001, /* set required minimum capacity % to the lowest value */
> +  };
> +
> +  INIT_IEEE1275_COMMON (&args.common, "call-method", 3, 2);
> +  args.method = (grub_ieee1275_cell_t) "ibm,client-architecture-support";
> +  rc = grub_ieee1275_open ("/", &root);
> +  if (rc)
> +    {
> +      grub_error (GRUB_ERR_IO, "could not open root when trying to call 
> CAS");
> +      return;
> +    }
> +  args.ihandle = root;
> +  args.cas_addr = (grub_ieee1275_cell_t) &vector;
> +
> +  grub_printf ("Calling ibm,client-architecture-support from grub...");
> +  IEEE1275_CALL_ENTRY_FN (&args);
> +  grub_printf ("done\n");
> +
> +  grub_ieee1275_close (root);
> +}
> +
> +#endif /* __powerpc__ */
> +
>  static void
>  grub_claim_heap (void)
>  {
>    unsigned long total = 0;
>
> +#if defined(__powerpc__)
> +  if (grub_ieee1275_test_flag 
> (GRUB_IEEE1275_FLAG_CAN_TRY_CAS_FOR_MORE_MEMORY))
> +    {
> +      grub_uint64_t rma_size;
> +      grub_err_t err;
> +
> +      err = grub_ieee1275_total_mem (&rma_size);
> +      /* if we have an error, don't call CAS, just hope for the best */
> +      if (err == GRUB_ERR_NONE && rma_size < (512 * 1024 * 1024))
> +     grub_ieee1275_ibm_cas ();
> +    }
> +#endif
> +
>    grub_machine_mmap_iterate (heap_init, &total);
>  }
>  #endif
> diff --git a/include/grub/ieee1275/ieee1275.h 
> b/include/grub/ieee1275/ieee1275.h
> index f53228703..b5c916d1d 100644
> --- a/include/grub/ieee1275/ieee1275.h
> +++ b/include/grub/ieee1275/ieee1275.h
> @@ -128,6 +128,14 @@ enum grub_ieee1275_flag
>    GRUB_IEEE1275_FLAG_CURSORONOFF_ANSI_BROKEN,
>
>    GRUB_IEEE1275_FLAG_RAW_DEVNAMES,
> +
> +  /*
> +   * On PFW, the first time we boot a Linux partition, we may only get 256MB 
> of
> +   * real memory area, even if the partition has more memory. Set this flag 
> if
> +   * we think we're running under PFW. Then, if this flag is set, and the 
> RMA is
> +   * only 256MB in size, try asking for more with CAS.
> +   */

#if defined(__powerpc__) ?

> +  GRUB_IEEE1275_FLAG_CAN_TRY_CAS_FOR_MORE_MEMORY,

#endif

Otherwise Reviewed-by: Daniel Kiper <daniel.kiper@oracle.com>...

Daniel